Learning Discrete-Time Markov Chains Under Concept Drift

Lecture 11: Discrete Time Markov Chains

where {Zn : n ∈ N} is an iid sequence, independent of initial state X0. If Xn ∈ E for all n ∈ N0, then E is called the state space of process X . We consider a countable state space, and if Xn = i ∈ E, then we say that the process X is in state i at time n. For a countable set E, a stochastic process {Xn ∈ E,n ∈ N0} is called a discrete time Markov chain (DTMC) if for all positive integers n ∈ ...

Discrete Time Markov Chains : Ergodicity Theory

Lecture 8: Discrete Time Markov Chains: Ergodicity Theory Announcements: 1. We handed out HW2 solutions and your homeworks in Friday’s recitation. I am handing out a few extras today. Please make sure you get these! 2. Remember that I now have office hours both: Wednesday at 3 p.m. and Thursday at 4 p.m. Please show up and ask questions about the lecture notes, not just the homework! No one cam...

Discrete Time Markov Chains 1 Examples

Example 1.1 (Gambler Ruin Problem). A gambler has $100. He bets $1 each game, and wins with probability 1/2. He stops playing he gets broke or wins $1000. Natural questions include: what’s the probability that he gets broke? On average how many games are played? This problem is a special case of the so-called Gambler Ruin problem, which can be modelled using a Markov chain as follows. We will b...

Lecture 10: Discrete Time Markov Chains

Discrete Time Markov Chains with R by

The markovchain package aims to provide S4 classes and methods to easily handle Discrete Time Markov Chains (DTMCs), filling the gap with what is currently available in the CRAN repository. In this work, I provide an exhaustive description of the main functions included in the package, as well as hands-on examples. Introduction DTMCs are a notable class of stochastic processes. Although their b...